Automatic dynamic contextual data entry completion system

ABSTRACT

A method, performed in a character entry system involves computing contextual associations between multiple character strings based upon occurrence of character strings relative to each other in documents present in the system, wherein the computing contextual associations involves i) identifying pertinent documents present in the system, ii) creating a list of character strings contained within documents in the system; and iii) creating an interrelationship between the character strings to contents of the system; in response to the user inputting a specified threshold of individual characters, identifying at least one selectable character string from among the character strings used in creating the computed contextual associations that can complete the incomplete input character string in context; providing the identified at least one selectable character string to a user for selection; and receiving, in the system, the user&#39;s selection and completing the incomplete input character string based upon the selection.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/040,470 filed Jan. 21, 2005 now U.S. Pat. No. 7,630,980, the entiretyof which is incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates to information processing and, moreparticularly, computer, cell phone, personal digital assistant, or othersimilar device-based text entry.

BACKGROUND OF THE INVENTION

In modem life, there are a number of devices, notably digital computersand multifunctional handheld units that involve data entry, typicallytext, including for example cellular phones and other devices likeorganizers and handheld computers. For all of these, one important useis the entry of linguistic items like words, phrases, and sentences. Forexample, a user may create an unstructured text document or mightformulate an email message or a short text message to be sent as an SMSmessage on a cell phone. In such cases, text entry may occur through useof a keyboard or stylus for some handheld computers or cell phones, etc.However, data entry can be difficult when the keyboard is relativelysmall as it is on a handheld cell phone, organizer or computer, or usesindividual keys for entry of multiple letters, text, especially when alarge number of characters must be entered. Similarly, with devicesemploying a stylus for text entry, entry of text can be slow andburdensome.

Automated word completion programs have eased the burden somewhat. Suchautomated word completion programs have appeared recently in a varietyof applications in a variety of devices. These programs are typicallybased on either predefined word suggestion lists (e.g. a dictionary) orare culled from the user's own most recently typed terms, the latteroften called MRU (i.e. “Most Recently Used”) programs. For example, theformer type of program is based on a pre-given word suggestion listbased on a dictionary augmented with information about which words aremore frequently used. If a user types the characters “su” in a document,then it might suggest “super” as the appropriate word completion basedon the fact that it belongs to the pre-given word suggestion list andhas a high frequency of use in general English. On the other hand, thelatter type of program would suggest a word completion based on theuser's own recently used words (e.g. “supreme” may be suggested to alawyer who has recently input “Supreme Court”). Such programs are oftenfound in web browsers for example and will suggest the most recentlyused “uniform resource locator” or URL (e.g. www.google.com when theuser types “www.g”) as characters are input.

A third type of program is able to detect that the user is in aparticular type of field (e.g. the closing of a letter) and will suggestword completions (e.g. “Sincerely” when the user types “Si”) based on amore limited “contextual” list. An extension of this is to maintain manyseparate word suggestion lists and allow the user to choose anappropriate list for each document the user creates. Other variantsallow users to actually insert entries manually into word suggestionlists (e.g. a name and address) or to maintain frequencies of word usageby a user and thus, rather than offering the most recently used word,offer the user's most frequently used words.

SUMMARY OF THE INVENTION

While the methods delineated above have many useful features, there isstill a lack of a true context based system that is dynamic andautomatic and thus, there is still much room for improvement when itcomes to data entry in such devices. Systems that maintain separate wordlists and allows the user to choose an appropriate list are contextualto some degree, but still have the drawback of requiring the user tomake a list selection each time, something that can become annoying fora user who typically creates several documents within the course of asingle day. Moreover, separate word suggestion lists are stillinefficient because they are not automatically generated but insteaddepend on the user's guidance and input.

The present invention combines certain features from existing techniquesbut goes significantly beyond them in creating a family of techniquesthat are automatic, dynamic, and context based as explained in greaterdetail herein.

The present invention involves a method, performed in a character entrysystem. The method is used for interrelating character strings so thatincomplete input character strings can be completed by a selection of apresented character string. The approach involves computing contextualassociations between multiple character strings based upon co-occurrenceof character strings relative to each other in documents present in thecharacter entry system, identifying at least one selectable characterstring from among the computed contextual associations that can completethe incomplete input character string in context (performed in responseto inputting of a specified threshold of individual characters), andproviding the identified at least one selectable character string to auser for selection.

The advantages and features described herein are a few of the manyadvantages and features available from representative embodiments andare presented only to assist in understanding the invention. It shouldbe understood that they are not to be considered limitations on theinvention as defined by the claims, or limitations on equivalents to theclaims. For instance, some of these advantages or features are mutuallyexclusive or contradictory, in that they cannot be simultaneouslypresent in a single embodiment. Similarly, some advantages areapplicable to one aspect of the invention, and inapplicable to others.Thus, the elaborated features and advantages should not be considereddispositive in determining equivalence. Additional features andadvantages of the invention will become apparent in the followingdescription, from the drawings, and from the claims.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates, in simplified form, a top-level flowchart for theautomatic completion of character input using contextual wordassociations;

FIG. 2 a illustrates a simplified flowchart for computing contextualassociations in one example implementation of the invention;

FIG. 2 b illustrates a simplified flowchart for the selection ofcontextual associations in an example implementation of the invention;

FIG. 3 illustrates an example documents versus words matrix used tocompute contextual associations with an example implementation of theinvention;

FIG. 4 a illustrates a simplified flowchart for computing contextualassociations in an alternative example implementation of the invention;

FIG. 4 b illustrates a simplified flowchart for the selection ofcontextual associations in an alternative example implementation of theinvention;

FIG. 5 illustrates an example matrix of pairs of words used to computecontextual associations in the alternative example implementation of theinvention; and

FIG. 6 illustrates an example set of word lists for a word completionexample involving the alternative example implementation.

DETAILED DESCRIPTION OF THE INVENTION

The present invention can be used with a variety of electronic devices.The minimum requirements for any such device are some means foraccepting textual input from a user, one or more processor(s) thatexecute stored program instructions to process the input, storage forthe data and the program instructions and a display or other outputdevice of some sort to make output visible or available to the user.Representative, non-exhaustive, example input devices can include, butare not limited to, a keyboard, a handwriting recognition system thatmakes use of a stylus, a touch pad, a telephone keypad, a pointingdevice like a mouse, joystick, trackball or multi-directional pivotingswitch or other analogous or related input devices. The storagepreferably includes non-volatile memory, and can also include volatilesemiconductor-based memory, electro-magnetic media, optical media orother types of rewriteable storage used with computer devices. If adisplay is used, the display may be small and capable of displaying onlytext or much larger and capable of displaying monochrome or color imagesin addition to text. If another output device is used, like a text tospeech converter, appropriate implementing equipment will be included.Although described, for purposes of clarity, with reference tokeyboard-type entry, it is to be understood that the present inventionis independent of the particular mode of, or device used for, text dataentry.

At the outset, it should be noted that, for the purposes of thisinvention, a “document” as used herein is intended to be a very generalterm covering one or more characters, whether alone or in conjunctionwith numerals, pictures or other items. A document's length can varyfrom a single “word” to any number of words and it can contain manytypes of data other than words (e.g. numbers, images, sounds etc.).Thus, ordinary documents such as pages of text are documents, but so arespreadsheets, image files, sound files, emails, SMS text messages etc.

As noted above, a “word,” for the purposes of this invention, can beconsidered to be more than a string of alphabetic characters, it mayinclude numeric and other symbols as well. Broadly, the inventionprovides contextual completion of character strings, where a characterstring includes not only alphabetic words but any other discretecollection of characters, symbols, or stroke based pictographs orideograms, for example, those used in languages like Chinese, Korean andJapanese, and thus can benefit from use of the present invention. Thus,although for simplicity the term “word” is used in the followingdiscussion, it should be understood to encompass any discrete collectionof characters, symbols or other stroke based representations ofcommunicative concepts, thoughts or ideas. Thus, the present invention,although described with reference to English, is independent of anyparticular language. It can be used for phonetic, pictographic orideographic languages when the characters, pictograms or ideograms usedtherein (or “stroke” components thereof) are considered “words” andthereby are intended to be incompassed by the terms “text” and“textual.” In some cases, an entire pictogram or ideogram will be usableas a “word” as described herein with entry of a component of thepictogram or ideogram, such as a defined stroke, being analogous toentry of a letter in English. Likewise, for simplicity in the followingexamples, the terms “typing” or “typed” are used to describe data entry.However, those terms should be broadly read to encompass any and allmethods of data entry, whether involving entry through use of akeyboard, a pointing or selection device, a stylus or other handwritingrecognition system, etc. They are not in any way intended to be limitedonly to methods that make use of a typewriter-like keyboard.

Examples of devices that can use and benefit from incorporation of theinvention therein range from large computer networks, where animplementation of the invention may be part of or an application on thenetwork, to small portable hand held devices of more limited orspecialized function such as cell phones, text messaging devices andpagers. Implementations incorporating the invention can be used toassist users in interacting with large databases by helping in the entryof search terms or in data entry. Other implementations incorporatingthe invention are particularly useful for portable devices, in which theinput device is limited by size and difficult to work with, because theautomatic completion of character string entries provides greaterbenefits in such devices. Still other implementations incorporating theinvention are particularly useful for devices used by those withphysical handicaps. In addition to the methods of character inputalready mentioned, devices intended for use by handicapped individualsmay rely on some type of pointing device to select individual charactersfor input. The pointing device may be controlled by movement of theeyes, head, hands, feet or other body part depending on the abilities ofthe particular individual. The present invention may also be used with“text” that is implemented in braille or other tactile representationsfor individuals with impaired vision.

In overview, in connection with the invention, words from one or moredocuments are associated, in either a fully or partially automated way,based on context. Context is derived from the co-occurrence ofindividual words in documents. In addition, the associations can bepre-computed and static or dynamic so they can thereby evolve andimprove with continued use.

For example, in an implementation of the invention, an associationbetween “finance” and “summary” may be generated but not one between“finance” and “sugar;” in this case, if a user has typed in the word“finance” followed by the characters “su,” then, based on theassociation, the invention will suggest “summary” as the appropriateword completion rather than “sugar.” Here, the word “finance” hasprovided the context that suggests the appropriate completion; ifinstead the user had typed “two spoons of” and then the characters “su,”and if an association had been generated between, “spoon” and “sugar”rather than “spoon” and “summary” then the invention would suggest“sugar” as the contextually appropriate completion. As more words areentered in the document, the contextual associations become richer.

The invention permits the use of different techniques for actuallycreating the associations. As a result, for purposes of understanding,two fully automated example techniques are described below with theunderstanding that semi-automatic implementation techniques areconsidered to be literally the same as the fully automated ones. Theautomatic or manual nature of a technique is, in most respects,independent of the invention because it relates more to the ease ofprocessing large amounts amount of text, not the technique itself.

The general approach is illustrated, in simplified overview, in FIG. 1with respect to a single document. The approach begins with a devicesuch as a personal digital assistant, cell phone, computer or otherdevice (100, 110, 120 or 130) which has documents (140) stored in itsmemory. These documents are used to create associations (150) betweenpairs of words or character strings within the document and use theseassociations to suggest word or character string completions (170) tothe user entering text (160) in a document. The associations among thewords or strings may be static or dynamic. With implementationsincorporating a more dynamic approach, as the user adds to a document orcreates more documents on the device, the associations are recomputed orsuitably augmented. This will alter the set of associations by eitheradding new associations, deleting existing associations or both. Thus,with implementations of the automatic contextual word completion systemhaving this “dynamic” aspect, the system evolves as the user adds to orcreates new documents and thus generally improves with use. Extensionsto these implementations further allow the device to impliedly track theuser's evolving interests.

Associations between words can be computed in a variety of ways and, asnon-limiting examples, two alternative automatic methods of doing so aredescribed.

In the first method, the first step is to assess the similarity of wordswithin one document or from one document to other documents that mayexist on the user's device. In this method, contextual associations arearrived at by grouping documents based on similarity and creating listsof words that are common to each group. There are many known methods toassess document similarity including the Jaccard, Dice or cosinecoefficients and the K-vec methods. For purposes of explanation, onesuch example similarity assessment method, based on treating documentsas vectors in a multidimensional space, is used, it being understoodthat, depending on the particular implementation, other similarityassessment methods can be used in addition to, or instead of, those usedin the examples described herein for practical reasons.

This example method is outlined in the flowcharts in FIGS. 2 a and 2 b.The method starts by creating a list of all the pertinent documents(200) on the device. From this list of pertinent documents a list ofunique words is created (205). An optional step, is to remove stop wordsfrom the word list (210). Stop words are described in greater detailbelow but include words like “the,” “at” and “in.” For each word in theword list, the number of times it occurs in each document is counted(215) and this number is stored in a matrix of documents vs. words(220). This matrix is used to calculate a similarity value (225) foreach possible pair of documents in the document list. The similarityvalue for each document pair is compared to a threshold value (230) andthose document pairs whose similarity value falls below the specifiedthreshold value are discarded (235). The remaining document pairs areused to group documents such that the similarity value of each possiblepair in each group is above a specified threshold value (240). Lists ofunique words from each group of similar documents are created (245).Words within each of these lists are contextually related. The steps ofthe example method to this point may be carried out independently ofuser text entry or, in implementations where the dynamic aspects of theinvention are utilized, carried out simultaneously with user text entry,so that the contextual associations are updated as the user enters morewords into the device.

Once at least an initial set of contextual associations exists, it canbe used at some point thereafter. The approach to use is as follows. Thedevice accepts character input from the user until a specified thresholdnumber of characters has been entered (250). Using the enteredcharacters, relevant word lists are identified (255). Due to theprocessing, the words within these identified lists are deemedcontextually related and thus, words in the identified lists having acorresponding initial character string matching the entered charactersare chosen (260) to be offered for selection by the user to complete thecharacter entry (265).

The above referenced process can be fully understood by way of thefollowing simplified example. To assess the similarity or dissimilarityof documents, one way of thinking of a document that contains one ormore words is as a bag or multiset of words. A bag or multiset is likean ordinary set in mathematics, a collection, except that it can containmultiple occurrences of the same element. For example, {book, cape,pencil, book} is a bag containing four words of which the word “book”appears twice. The order of occurrence of elements in a bag does notmatter, and could equally be written as {book, book, pencil, cape}.Also, any bag can be converted to a set just by dropping multipleoccurrences of the same element. Thus, the example bag above, whenconverted to a set, would be {book, cape, pencil}. To create the bag ormultiset, the contents of a document with the exception of numbers whichare a special case are stripped of all internal structure (e.g.syntactic structure, punctuation etc.) including all nonlexical itemslike images, sounds etc. The resulting stripped document would be a bagor multiset of words as described above which may also include numbersand in which some words may occur multiple times. For a user who has adevice with a number of stored documents, each pertinent document issimilarly stripped down to form bags and the mathematical union of thesebags can be taken to form a larger bag.

As a side note, optionally, a certain class of words, typically called“stop words,” are removed from such document-derived bags. Stop wordsare words like “the,” “of,” “in” etc. and are removable because theyusually are not very informative about the content of the document. Stopwords, if removed, can be removed from the bags either before or after amathematical union of the bags is made, as the end result is the same.Typically stop words are identified in a list which can be used for theexclusion process. Since the stop word removal process is well known itis not described herein. In addition, in some implementations where astop word list is used, the list may be editable so that additionalwords can be defined as “stop words.” For example, otherwise non-trivialwords that are trivial in the particular context because they occur toooften in that context (e.g. words like “shares” in stock relatedgovernment filings).

By way of simplified example (FIG. 3), if the user has just twodocuments on a device: “d 1” (306) made up of “an apple, apple cider andan orange” and “d2” (308) made up of “a paper apple” then, eachcorresponding bag is {apple, cider, apple, orange} and {paper, apple}.Their union is the larger bag {apple, cider, apple, orange, paper,apple} and a set for the bag would be {apple, cider, orange, paper}.

A matrix (300) is then formed with for example, each element in the setof words derived from the documents on the user's device listed alongthe columns (302) of the matrix and each document itself (symbolized insome way) along the rows (304) of the matrix. In the cell correspondingto the intersection of a document “d” with a word “w,” the number oftimes “w” occurs in “d” is entered (318). For the simple example above,as shown in FIG. 3, for the cell corresponding to the intersection ofthe row for the first document “d 1” and the column for the word “apple”a “2” (318) is entered since it occurs twice in document “d1.” Thisoccurrence frequency information is obtained from the document bags. Ifa word does not occur in a particular document at all, a zero is enteredin the corresponding cell. Note that depending upon the number ofdocuments and the number of words, the size of the matrix can beexceedingly large. Moreover, there is no significance to whether rowslist documents and columns list words or vice versa—the contents of therows and columns could be exchanged without affecting the invention.

Once the matrix is created, each document is treated as a vector in amultidimensional Euclidean space, with the number of dimensions beingthe number of words or columns of the matrix. Thus, the simplifiedexample of FIG. 3, each of documents d1 and d2 can be treated as a fourdimensional vector since there are four elements in the correspondingset {apple, cider, orange, paper}. Notably, by using this approach, thewords can also be listed in any order in the columns, for example, inorder of first or last occurrence, randomly, alphabetically, by numberof letters, stroke type, stroke count, etc., in forward/increasing orreverse/decreasing order.

The similarity between d 1 and d2 is then estimated by measuring thecosine of the angle between vectors d1 and d2. The higher the cosinevalue, the greater the similarity, the lower the cosine, the greater thedissimilarity. If the value is the maximum possibility of “1,” thevectors are fully aligned in multidimensional space and the two areidentical. On the other hand, if the cosine reaches its minimum of “0,”the vectors or documents are completely dissimilar (i.e. they areorthogonal). For the absolute intermediate cosine value of “0.5,” thedocuments are roughly as similar as they are dissimilar.

This method can thus be extended where more than two documents exist, bypairing the documents in all unique combinations and performing the samesimilarity analysis on all pairs of documents that are pertinent (i.e.under consideration) from the device of the user. Note that it is likelythat every “document” on the user's device may not be “pertinent” forpurposes of this invention, for example, it will likely be desirable todifferentiate between the user's documents and documents that may bepresent for other reasons (e.g. programs, system configuration filecontents, system data, etc.) and thus irrelevant. For purposes ofunderstanding the invention, it should be assumed that non-pertinentdocuments have been excluded in some manner, whether manually orautomatically.

As to all of the pairs, cosine values are computed and a threshold valueis chosen. The threshold value represents a divider in which all cosinevalues above the threshold are accepted as indicating meaningfullysimilar document pairs and below which the document pairs aresufficiently dissimilar so as to be considered not meaningfully related.In other words, a range of values is taken and on the basis of thosevalues, an arbitrary division is formed to differentiate, and deem eachdocument, meaningfully similar or dissimilar relative to each otherdocument.

In application, the threshold value mentioned above is initially apreset parameter, the value of which may be different for differenttypes or classes of target users. However, in some implementations, thethreshold can be adjusted, whether over time through use or because ofchanges in the quantity or contents of the pertinent documents, andeither by program modification or through user action. In otherimplementations, it can be adjusted automatically or by implication asis discussed in greater detail below.

In all cases, what is obtained, either through the vector space methoddescribed above or through other analysis approaches, such as Jaccard,Dice or cosine coefficients, the K-vec methods or some other method, isa judgment of similarity and dissimilarity of document pairs in thepertinent document collection or set.

This similarity judgment is then used to form groups of documents, eachgroup of which contains only documents that are sufficiently similar toone another when compared in a pair wise fashion. Note that therelationship of similarity is reflexive and symmetric, but it is notnecessarily transitive. This means that the groups may not be disjointi.e. the same document may belong to more than one group, particularlyin implementations where a document need not be sufficiently similar toevery other member of the group, but only some specified portionthereof. In other words, as a result of the grouping, two or more groupswill be formed wherein each document is meaningfully similar to at leastsome specified portion of the other documents in that group. In general,each group will have the following characteristics: each document in thegroup will be sufficiently similar (i.e. be above the threshold) withrespect to at least some specified portion of the group (in practice,likely preset at well more than 50% if stop words are not removed), eachgroup will be differentiable from any other group by at least onemember, and each group will not be a simple subset of any other group.

In addition, some implementations may involve large numbers of nearlyidentical documents such that the common information is not meaningfulbecause it is ubiquitous and the differentiating information may be toounique relative to each other to be of interest. One such example, wouldbe certain filled out forms. In such a case, a second very highthreshold can be used as a “screen” whereby copies of documentsexceeding this threshold are separately grouped, the mathematicalintersection of the documents are used to form a “stop word”—like list,the intersection-based stop word list is then used to strip the copiesof those words, and then the copies are used in place of the actualdocuments in the overall grouping process. In this manner, the ubiquitycan be handled without sacrificing the potential contextual relevance ofthe unique information to other pertinent documents.

Once groups of similar documents have been made, lists are formed foreach group containing all non-stop words in the union of the setscorresponding to each document in the group. As with the lists above,these lists may be arranged in any of a variety of possible orders (e.g.alphabetical, first/last occurrence, etc.). In addition, the lists candynamically change, for example, by repeating the above processing (inwhole or part) as time progresses and the user creates/modifies/deletesdocuments.

As a result, the lists created as described above are essentially listsof all words associated with each other i.e. once the lists have beencomputed, the associations mentioned earlier have been computed as well.By way of simple example with respect to the two documents d1 and d2introduced above, if a list contains the words {apple, cider, orange}then it can be assumed that these words are associated with each other.The word “paper” from d2 above is not on this list and so is notassociated with “apple,” “cider” or “orange” through this list, but maybe associated with apple through another list “d3” that contains {apple,orange, knife, paper, plate}. It may also happen that there also isanother list “d4”, based on another group of documents, that contains{orange, paper, peel} in which case “orange” and “paper” would beassociated with each other through that list as well as through “d3”.

Thus, based upon the above, if a user were to type “apple” followed bythe characters “ci,” then the program instructions implementing thisaspect could check the lists, find that “apple” and “cider” areassociated through a list, and thus offer “cider” to the user as apossible completion of the incomplete word. Similarly, if a user were totype orange followed by “p”, then “paper” and “plate” would be relatedto “orange” through list “d3” and “paper” and “peel” through “d4”.Depending upon the next letter entered, the “contextually related”contents of one of the lists would be excluded or the one word in common“paper” would be provided as the “contextually related” completion word.

In the first of the immediately preceding examples, a single word“apple” is used as the basis for choosing “cider” as the possiblecompletion. Whether a single word, multiple words, sentences or evenlonger segments of text are used as the basis for choosing the possibleword completion is dependent on the particular implementation. Forsimple devices, limitations on memory and processing resources willlikely dictate that no more than a few words of prior user input be usedas the basis for choosing the word completion. In other implementationsthis could be a user selectable parameter, a preset parameter in thedevice or could be adjusted dynamically by the program.

At this point it should be understood that document similarity is basedon what words, if any, are shared by each of the pertinent documents andin what frequencies when similar documents are grouped and used to formlists. Thus, by grouping “similar” documents, an assumption is made fromthe similarity analysis that words that belong to the same broad subjectmatter are being collected—i.e. “similar” documents are necessarilylikely to contain words that pertain to the same subject matter. Hence,the derived associations are likely to be more relevant to a user who istyping a document on that subject than other alphabetically similar ormost recently used, but non-associated, words.

While the formation of groups is based on the calculated similarityvalue between pairs of documents, the details of how the similarityvalue is used can vary depending on the particular implementation of theinvention. For example, as defined above, all document members of agroup must have a similarity value above the set threshold value whencompared to some portion of the other member of the group. However, that“portion” can be more or less restrictive. The most restrictivedefinition requires the portion be all members of a group and leads tothe largest number of groups and development of word lists that tend tobe shorter but the contextual associations among words in the list tendto be stronger. The least restrictive definition of a group is thatanyone document of the group need only have a similarity value above thethreshold value when compared with anyone other member of the group.Although it will form the fewest groups, this definition Will, forpractical purposes, never be used. In practice, if a preset threshold isspecified, it will likely be set so as to have a reasonable number ofgroups while having meaningfully usable relationship lists. As currentlycontemplated, an initial threshold within the range of 50% to 90% wouldbe used as a reasonable empirical starting threshold. Notwithstandingthe foregoing, the threshold can be initially set to a value outsidethat range or can, in some implementations, ultimately end up outsidethat range either through evolution or affirmative changing of thesetting. Regardless of how restrictive the definition of a group is,whenever a new document is found to have a similarity value that isbelow the set threshold when compared to all of the existing documentson the device, that document becomes the first member of a new group.When less restrictive definitions of a group are used, groups, and theirassociated word lists, become larger and the strength of the contextualassociations between words in the list becomes weaker. Thus, thethreshold will clearly be a parameter that, even with its initial value,will vary with the particular implementation of the invention.

It will now be apparent that the threshold value for similarity plays animportant role in the formation of groups and thus the lists ofcontextually associated words. The ability to distinguish betweenclosely related documents will also increase as the threshold value isset higher. Therefore a high threshold value may be appropriate for anattorney who specializes in real estate matters and who createsdocuments, such as sales contracts or leases, which have a high degreeof similarity, while a low threshold value may be appropriate for aneditorial writer who produces documents on a wide variety of generalinterest topics.

The threshold value may be set by the designer of the device dependingon the intended use, selectable by the user or set dynamically by theprogram implementing the invention. One alternative implementationmethod which, in effect, dynamically sets the appropriate thresholdvalue involves specifying the maximum number of groups that can becreated. By doing so, the system will be required to start at arelatively high threshold value and, if the resulting number of groupsexceeds the specified maximum, it will adjust the threshold downwarduntil the maximum number is not exceeded. If no more than five groupsare specified then the threshold value will be lower than if no morethan 20 groups are allowed. The program implementing the invention canbe further optionally configured to, as documents areadded/changed/deleted, monitor the number of groups and adjust thesimilarity threshold value so that the number of groups is continuallywithin the specified limit. Additionally, the limit can be madevariable, for example based upon the number of documents in the system.In this manner, once a greater threshold number of documents isexceeded, a new maximum can become effective so as to allow for moregroups and maintain a meaningful granularity.

Another optional additional method that may be used by the programimplementing the invention, is to dynamically set the similaritythreshold value by monitoring the performance of the word completionsystem. For optimal performance of an automatic word completion system,the system must strike a balance between the case of rarely offering aword completion but the completion offered is always appropriate, andthe case of always offering a word completion but the completion offeredis often inappropriate. By monitoring how often the system offers apossible word completion to the user and how often the user accepts theoffered word completion, the program can determine if the strength ofthe contextual associations is too high or too low. If the user isfrequently having to discard suggested word completions, the program canincrease the strength of the contextual associations by raising thethreshold value or, as discussed above, making the definition of a groupmore restrictive or both.

Yet another optional use of the similarity threshold value is to rankpossible word completions so that the most likely word completion isoffered to the user first. The ranking is accomplished by computing theword associations using different similarity threshold values. Inimplementations of the invention where the user is offered multiplepossible word completions, words that are contextually associated withthe partial input based on use of a high threshold value would beoffered at the top of the list, while those words that are onlycontextually associated based on the use of a low threshold value, wouldbe offered at the bottom of the list.

To further illustrate how different documents may be grouped, consider acollection of four documents, a draft of the Declaration ofIndependence, the signed version of the Declaration of Independence, theUnited States Constitution, and an article reviewing the foundation ofUnited States constitutional law. In the case of a user who hasdocuments covering widely different topics such as the history ofbaseball, art and literature a relatively low similarity threshold maybe set so that there are only four groups of documents. The Declarationof Independence, Constitution, and constitutional law article wouldlikely all be in one group as they are clearly related documents butwould not be related to documents on baseball, art or literature. In thecase of another user who happens to be an attorney and who has onlylegal documents on his device the similarity threshold value would haveto be set higher so that different classes of legal documents could bedistinguished. If the threshold value were set too low, then the factthat words such as “liberty” are present in all four of the documents inour example collection, would be enough for all four documents to beplaced in the same group. With a higher threshold value, words like“creator,” which only appears in the Declaration of Independence, and“Senate,” which only appears in the Constitution and the constitutionallaw article, would result in these documents being placed in twodifferent groups, one on the Declaration of Independence and one on theConstitution.

A second example method for computing contextual associations betweenwords, suitable for use with the present invention, is outlined in theflowcharts of FIGS. 4 a and 4 b. As in the first example method, thesecond method begins by creating a list of pertinent documents on thedevice (400) and creating a list of unique words from these documents(410). Again, an optional step is the removal of stop words from theword list (420). For all possible pairs of words from the list, thefrequency of co-occurrence within a unit of text is counted (430) andthis result entered into a matrix (440). The particular “unit” that ischosen will be a parameter to be decided based upon various factorspresent for the particular implementation. Generally, if the documentsbeing considered are fairly long (such as a thesis, book or report),then a paragraph is typically considered as the appropriate unit; if thedocuments being considered are fairly short (e.g. like an SMS message ona cell phone or an e-mail), the whole document may be considered as aunit. Other “unit” options are, for example, the sentence, sections of adocument, chapters, pages columns, etc. The unit parameter may beinitially fixed, permanently fixed, modifiable or dynamic. In fact, insome implementations, the unit may even be allowed to change based onthe length of the document and the portion involved. For example, forpatent documents, the appropriate “unit” for the abstract might be theparagraph, for the claims it might be the line or sentence, and for thedetailed description it might be, for example, a section, column,paragraph or sentence.

The rows and columns of the matrix are lined with words from the wordlist. As before, the interchange of rows and columns changes nothingwith respect to the invention. The matrix containing the co-occurrencedata is used to identify word pairs that are contextually associated,based on their frequency of co-occurrence. The steps of the method tothis point can be performed independently of user text entry or, inimplementations where the dynamic aspect of the invention are realized,these steps can be performed simultaneously with user text entry so thatthe contextual word associations are updated as the user enters new textinto the device.

Once the contextual word associations have been determined, the deviceaccepts character input from the user until a threshold number ofcharacters are entered (460). Words in the matrix that match the enteredcharacters are identified (470) and the identified words that arecontextually related are chosen (480) and offered for selection by theuser to complete the character entry (490).

Note that, as described above, the matrices in FIGS. 3 and 5 have beenfilled with direct co-occurrence information. However, this is notrequired because there are many other variants of this approach that canbe used to the same, better, or in some cases less, effect Gust as thereare many different ways of measuring document similarity) depending uponthe particular implementation and documents involved. Thus, the rawinformation need not be used as is but may, for example, be normalizedin some way, ranked, and so on.

Optionally, once the co-occurrence matrix is prepared, it can bereorganized so it is ordered, for example, according to whichco-occurrences are higher relative to each letter of the alphabet.

In an example system implementing this method of the invention, and inreference to the matrix (500) in FIG. 5, when a user types “finance”followed by the characters “su,” the system would check theco-occurrence matrix in the row corresponding to “finance” (502) to seewhich words starting with “su” have the highest co-occurrence score.Thus, if “summary” has a higher score than “sugar,” the system wouldoffer “summary” rather than “sugar” as the appropriate completion word.

In yet another implementation, variants of the above two methods can berun together as a cross-check or to get more suggestions. However,limitations of speed and memory may render this dual method approachimpossible, impractical or overly costly for some implementations.

It is to be understood that, while the two approaches have beendescribed in terms of bags and sets, because they are conceptually themost appropriate mathematical description, the actual implementations ofthese data structures can vary (e.g. they can be lists rather than bagsor sets), the particular data structure used to accomplish the resultbeing unimportant to the invention. Optionally, in some implementationsthe words in the lists are further processed to reduce them to stemmedform. That is, they may not be the full words but rather words withtheir endings suitably modified, for example, by one of the manyavailable stemming algorithms. This would enable the program to handlewords like “finance,” “financing” and “financial” or “remain,”“remainder” and “remaining” in a uniform manner so that the user will beoffered the same suggestions independent of the particular variant ofthe contextual words he has typed. However, stemming is optional. Theremay be advantages in not stemming words in some implementations, becausethe system can differentiate among them if the words are not stemmed andthereby offer one variant on one occasion and another on anotheroccasion.

The number of initial characters that are needed in order for the systemto make a suggestion to complete the word is also a parameter that canbe preset or set by the user depending on the implementation. As withcompletion algorithms of the prior art, in most implementations, this is2 or 3 characters. A second parameter that needs to be defined for aparticular implementation is how long the full word needs to be in orderfor the system to even make a suggestion. As contemplated, the minimumword length is 5 characters or strokes although minimum lengths of 2 or3 characters (or strokes) can be used. That is, if the characters “su”were typed, the system may not suggest “sure” even if it happens to bethe best association because it is too short a word. On the other hand,“summary” and “sugar” are acceptable candidates as they meet the lengthrequirement. This minimum length is also a parameter that will vary withdifferent implementations and possibly different users. Once theassociations have been computed the resulting information is used tosuggest a word completion.

Having described component aspects in simplified and theoretical form,particular example implementations are now discussed with reference tothe example word lists of FIG. 6 (600).

Assume groups of documents to be used have been created, stop words havebeen removed, the non stop words that appear in each group collected andalphabetized to form the series of lists of FIG. 6.

Example 1

In one simplified example implementation, a user types “finance”followed by the characters “su.” The system checks that “finance” is nota stop word (if it was, no word would be suggested) and then checks allthe word lists for the word “finance.” It would find that it just occursin word list #1 (602). Next, it checks all words in that list (or lists,if there is more than one where “finance” occurs) that begin with “su”and it finds the word “summary.” As a result, the system offers tocomplete the user's word with “summary.” The user can opt to accept thatsuggestion or not by the appropriate action, in this case pressing anappropriate key (e.g. the “Enter” key). Note, that as a result of theabove processing, “sugar” is not on this list and thus it is judged notto be a contextually relevant suggestion.

Example 2

In another simplified example implementation based upon FIG. 6, the usertypes “two spoons of” and then the characters “su.” Again, the systemlooks for the words “two” and “spoon” (the plural ending can be assumedto be stemmed in real time if stemming is used) on the word lists.Because, in this example, the word “of” is on the stop word list it isignored. The system finds “two” in word lists 2 (604) and 3 (606) andfinds “spoon” in word list 2 (604). As a result, it looks for wordsbeginning with “su” in all common lists first, i.e. in this example,word list 2 (604). It finds “sugar” in word list 2 (604) and offers itas a suggestion to the user, for acceptance or rejection. In this case,although word list 3 (606) contains the word “summary” which could havebeen a possible suggestion, it was eliminated because “summary” isassociated with “two” through word list 3 (606) but it is not associatedwith “spoon” on any list.

Example 3

In a further simplified example implementation based upon FIG. 6, a usertypes in “one” followed by the characters “ch.” Now “one” occurs in wordlists 2 (604) and 3 (606). There is no other information to find acommon list or to eliminate any list containing “one” as was done with“summary” in the previous example. In this case, both lists areconsidered and searched for words starting with “ch.” There are two:“chili” in list 2 (604) and “chapter” in list 3 (606). Depending on theimplementation, both suggestions are offered to the user for acceptanceor rejection or the system can wait until the user types a thirdcharacter. Note that, if the former occurs, optionally, the system canoperate to ignore further character entry for that word or represent aselection based upon the next character entered. This can occur onlyonce or interactively, the former allowing for entry of words not in anylist without annoying prompts, the latter allowing for discerning amongmultiple words with similar starting letters like “heat,” “heavy,”“heather,” “heaven,” hearing” and “health” which all share an initial“he a” or different words having a common root like “finance,”“financing,” “financial,” and “financially.” In the example, a thirdletter is waited for, so when the characters “cha” are typed in, “chili”is eliminated as a possible completion and just “chapter” is offered.

Even when there are multiple words that have been typed as in Example 2,if there are no common lists or if there is more than one list incommon, similar options can be employed, i.e. either all of thesuggestions are offered or wait until additional characters are entered.If this is done interactively, eventually the system will most likelyeliminate possible options until just one or a few can be offered to theuser. As with other aspects described herein this can be preset orset/modified by the user, depending upon the particular implementationinvolved. Some users find it annoying to have too many suggestions whileothers find it acceptable. Of course, it is also possible that the wordsalready typed do not belong to any word lists in which case, in oneexample implementation, no suggestions are offered. Alternatively, insuch cases, a prior art approach such as providing a word from astandard online dictionary or a word from a most recently used list canserve as a fallback. These again are implementation-specific decisions.

While the above simplified examples reflect the kinds of situations thatcan arise, by no means do they exhaust all the possible permutations andcombinations of situations that might occur in practice in a morecommercially suitable implementation. For such systems additionaloptional extensions can be employed to make the approach faster, morerobust, or otherwise more suitable.

One such optimal extension uses frequency information (i.e. the numberof times the word occurs in the set of documents under consideration)stored with or otherwise associated with the words in word lists so thatwhen there is more than one choice available to offer to a user,frequency ranking can be used to specify the offering. This is nowdescribed with respect to a simplified example matrix presumed generatedfrom a group of documents using a technique described above illustratedin FIG. 5.

The matrix of FIG. 3 is similar to that of FIG. 5 except that only halfof the matrix is filled out because the information in the lowerdiagonal half would be symmetric to the information in the upper halfand it was created from a slightly different group of documents. In theexample of FIG. 5, the diagonal entries are left blank but it should beunderstood that they can alternatively optionally be filled with plainfrequency information (e.g. the number of times a word occurs in adocument).

Example 4

In an example system incorporating this simplified exampleimplementation which uses the co-occurrence matrix of FIG. 5, a usertypes “finance followed by the characters “su.” The system looks up“finance” in the rows (602) and finds the co-occurrence scores of thewords starting with “su” in the columns. These are “summary” and “sugar”and their scores are 12 and 1 respectively (identified at theintersection of the two in the matrix (500)). Since “summary” has thehigher score, it is offered as a suggestion to the user.

Example 5

In another example system incorporating this simplified exampleco-occurrence matrix the user types “two spoons of” and then thecharacters “su.” Now the word “two” (508) has scores of 3 and 7respectively for “summary” and “sugar” and “spoon” (516) has scores of 0and 14 for “summary” and “sugar” respectively. Using any of a number ofknown possible ways for combining scores to produce a unique or at leasta ranked outcome a preferred or selected result is obtained and thecorresponding word presented. Clearly, in this example, any reasonableformula should produce “sugar” as the preferred outcome and offer it asa possible completion to the user or alternatively offer bothsuggestions with “sugar” optionally ranked first.

Example 6

In a further simplified example implementation involving theco-occurrence matrix of FIG. 5, presume a user has typed “one” followedby “ch.” The two possible completions as seen from the table above are“chapter” (512) with a score of 6 and “chili” (514) with a score of 8.In this implementation, based upon the scores, the higher score may bechosen or, since the scores are relatively close to each other, a rankedlist may be offered. If the two scores had been identical the order ofpresentation can be based upon, for example, the alphabetical option,the most recently used option or some other such variant can be used tospecify presentation order. Alternatively the system can be set up forcases where the scores are the same to wait, without offering anysuggestion, until the user has entered one more character(s) (e.g.“cha”) in which case, in this example, “chapter” would be theunambiguous choice.

In examining the examples above, it is clear that there will often becases where there will be more than one possible completion to offer theuser. In some implementations, the system will not offer the user apossible completion until the user has entered sufficient characters sothat an unambiguous choice can be made. In other implementations, thesystem will rank the possible completions to be offered to the userbased on the contextual associations that have already been computed.For example, words with a higher co-occurrence score, words from morefrequently or recently accessed lists, or words from lists that have thegreatest similarity, can be offered first. Still further, in someimplementations, the rankings can be dynamically set in that they changeas the user enters more characters (whether in that word or during thecourse of document creation).

In addition to the contextual and automatic aspects which have beenillustrated so far, there is also an optional dynamic aspect that can beincluded. Where such an aspect is used, the user continues to createmore documents of different types, the system re-computes theassociations of words for example, through the methods described above.This allows the system to keep evolving and improving as it has more andor new data to work with. In the case where a co-occurrence matrix iscreated by pooling all of the unique words from all of the pertinentdocuments, the option of updating the co-occurrence matrix in real time,even as the user is creating a new document, is available.

Notably, the above examples have implicitly dealt with unstructureddocuments (like emails, SMS text messages, word processed documents,etc.) and not with structured documents like forms, letters andspreadsheets. Advantageously, such documents can also be included in theclass of documents useable with the invention. Moreover, the structureitself can additionally provide usable contextual clues. For example, ina letter, the closing usually contains words like “Very Truly Yours” or“Sincerely;” in such a situation, as a special case related to that usedin the prior art, the system could suggest special words like“sincerely” from a special list because of the location in the document.Similarly, certain words may always appear capitalized (e.g. like theuser's name and address) and these special words can once again behandled in a special way. In some implementations the word liststhemselves can be further extended so as to contain, as needed ordesired, a separate field for storing special information about the wordsuch as capitalization, formatting (bold, italics, etc.), hyphenation,inclusion of special characters, accents on certain characters etc.Since the storing of such information, per se, is known and in use in,for example, certain word processors and it is readily implementablebased upon that knowledge, for brevity it is not described herein.

As to treatment of numbers, in any given document there may be specialnumbers like the current date and year that can be offered ascompletions in the context of a letter or such numbers can be treated as“stop words.”

Advantageously the above techniques can further be extended to allow forphrase completion. One way of doing this is to find phrases andso-called collocations by using standard frequency-based techniques on aphrase, as opposed to a word basis and then apply the methods describedabove to these phrases. In addition, an advantage of the techniquesdisclosed herein is that they can be combined with existing prior arttechniques like Most Recently Used (e.g. “MRU”) techniques. For example,one way to use a combination of the present invention and the prior arttechniques is to narrow down the possible suggestion list using one ofthe above methods and then choose one or rank the remainder by an MRUtechnique. Another is to use regular dictionaries in the absence of anysuggestion the system might make. Finally, the order of words on a wordlist generated by the vector space method can be allowed to changedynamically based on the use the user makes of suggestions he receives.

Note that when a user acquires a device like a computer, cell phone orpersonal digital assistant for the first time, there are not going to beany user-created documents on the device to which the methods describedabove can be applied. To allow for this, until there is some history ofdocument creation, one or more of several options exist. One of theconventional prior art techniques could be used until a sufficientnumber of documents exist. Alternatively, manually created word lists orco-occurrence matrices of a generic type, i.e. that a multiplicity ofusers are likely to find useful, could be used; or more specialized wordlists based on different types of vocabularies or uses (e.g. forbankers, lawyers, businessmen, doctors, domestic uses etc.) could becreated by the provider or others. In general, such approaches would betemporary until the user has his or her own document-based history atwhich point the existing lists or co-occurrence matrices can either beaugmented, modified or completely dropped in favor of new word lists andmatrices generated by the user's own documents. These same approachescan also be used in the situation where there are documents on thedevice but the user is starting a new document and there areinsufficient words entered to establish associations.

An optional extension of the methods which uses the text being generatedas dynamic data will now be described, This method can be used, in someimplementations, to augment any of the approaches described above.

Consider a user who is typing a somewhat long document (for example,more than three paragraphs, although this is again a variable parameterthat can differ in particular implementations). Suppose they havealready typed a few pages (say two pages) and now they are on the thirdpage. The data entered so far can be used in any of two ways. One, asdescribed above, where it provides the context to the next word beingtyped. Another, as dynamic data, in which the first two pages are usedas data from which to create an auxiliary co-occurrence matrix of thetype discussed above dynamically (i.e. “on the fly”). This matrix canthen be used, as before in conjunction with either the word lists basedon document groups of the first method described above or with theco-occurrence matrix of the second method described above. This approachcan provide additional advantages because text in the document the useris currently working on will likely be more relevant to the wordcompletions that can be offered to the user than those in the lists.This dynamically created matrix of co-occurrences is either continuallyupdated on the fly, or periodically as the user continues to type—forexample after each paragraph or page is completed. Alternatively,instead of using a co-occurrence matrix, a plain frequency matrixcontaining the number of occurrences of each word in the document beingtyped could be used.

In instances where it is desirable that the results derived from thisdynamically-created data be combined with results derived from theoffline data, the intersection of the results can be taken and further,it can be optionally ranked in some way. Alternatively or in case of anempty intersection, the union can be taken or one set of results givengreater priority.

When a full keyboard is not available on the device (say a cell phone)there are currently methods (e.g. T9) which allow the user to enter, forexample, the numbers 7 and 8 which correspond to the multiple groups ofletters p, q, r, s and t, u, v respectively. A method like T9 offers theuser an alphabetically ranked list of acceptable combinations (e.g. “pu”and “su” are acceptable, “pt” is not) based on dictionary entries. Insuch a situation, the computations described in the foregoing exampleinvolving any of the possible approaches would still occur, but not justto, for example, “su” but also to, for example, “pu.” Presumably, if theuser were typing something like “finance su/pu” then, just ascompletions like “sugar” would not be acceptable, completions like“pushing” would also not be acceptable under the same logic. If itturned out that that completion was, in fact, relevant under any of themethods above then multiple possible completions would be offered andthese would be dealt with as before (e.g. by a ranked list).

Finally, it is to be understood that various different variants of theinvention, including representative embodiments and extensions have beenpresented to assist in understanding the invention. It should beunderstood that such implementations are not to be consideredlimitations on either the invention or equivalents except to the extentthey are expressly in the claims. It should therefore be understoodthat, for the convenience of the reader, the above description has onlyfocused on a representative sample of all possible embodiments, a samplethat teaches the principles of the invention. The description has notattempted to exhaustively enumerate all possible permutations,combinations or variations of the invention, since others willnecessarily arise out of the use of particular hardware or software orthrough specific types of applications in which the invention can beused. That alternate embodiments may not have been presented for aspecific portion of the invention, or that further undescribed alternateembodiments may be available for a portion of the invention, is not tobe considered a disclaimer of those alternate embodiments to the extentthose undescribed embodiments also incorporate the minimum essentialaspects of the invention, as claimed, or an equivalent thereof.

1. A method, performed in a character entry system so that incompleteinput character strings input by a user interacting with the characterentry system, that are part of a series of input character strings whichestablish a context for the incomplete input character string, can becompleted by a selection of a presented character string using an inputdevice connected to the character entry system, the method comprising:computing contextual associations between multiple character stringsbased upon occurrence of character strings relative to each other indocuments present in the character entry system, wherein the computingcontextual associations comprises: i) identifying pertinent documentspresent in the character entry system, ii) creating a list of characterstrings contained within documents in the character entry system; andiii) creating an interrelationship between distinct character strings inthe list using their occurrence in the documents of the character entrysystem; in response to the user inputting a specified threshold ofindividual characters using the input device, identifying at least oneselectable character string from among the character strings used increating the computed contextual associations that can complete theincomplete input character string in context; providing the identifiedat least one selectable character string to a user in a manner suitablefor selection by the user using the input device; and receiving, in thesystem, the user's selection and completing the incomplete inputcharacter string based upon the selection.